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© Motion compensating field interpolation method using a hierarchically structured displacement 
estimator. 

© A motion compensating field interpolation method is presented, which allows to interpolate several fields 
between every two transmitted fields of a digital television sequence. Using the model of translatorily displaced 
objects a hierarchically structured displacement estimator is applied to cope with relatively large displacements^ 
It provides a displacement vector with integer components for each picture element of the fields to be 
interpolated. A change detector is used to assure zero displacement vectors in unchanged areas. A two- 
coefficient spatio-temporal filter interpolates each picture element of the fields to be interpolated. By experimen- 
tal evaluations the model of pure translatorily displaced objects has proven to be sufficient to approximate many 
kinds of motion.- The presented interpolation technique allows the unblurred reconstruction of omitted fields of 
television sequences, widely preserving the natural impression of motion. 
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Motion compensating field Interpolation method using a hierarchically structured displacement es- 
timator - 



Background of the Invention: 

The present invention relates to a method for a motion compensating field interpolation as wen as the 

5 use of this method. Such a method is particularly suitable for the reconstruction of omitted fields of 
television sequences. . ~ 

The problem of interpolating fields in digital television sequences arises in the case of field frequency 
conversion. In source coding applications fields have to be reconstructed, which are dropped in the coder to 
aljpw.a transmission of television signals in digital channels with very low transmission bit rates. The CCITT 

w Study Group XV is currently investigating television codecs with 384 kbit/s transmission rate. The aim is to 
provide video conference services, using the Integrated Services Digital Network ISDN. In order to achieve 
this data rate it seems to be necessary to reduce the number of transmitted fields in addition to known 
source coding techniques. A recently presented work on this subject uses a field subsampfing by a factor of 
4:1 combined with a hybrid coding algorithm [1]. Then, at the receiver three fields have to be interpolated 

75 between every two transmitted fields. 

The problem is to generate a field at a given temporal position between two successive fields of a 
television sequence. Since the temporal luminance changes are oftentimes caused by moving objects, a 
simple field repetition technique, which inserts the nearest available field, yields jerkily moving objects [2], 
Another simple approach is linear interpolating by temporal filtering [3]. In this case for each picture 

20 element to be interpolated a weighted average of the corresponding picture elements at the same spatial 
position is calculated. This interpolation technique is able to adapt to luminance changes which, e.g. are 
only due to illumination changes of the scene contents. However, blurring becomes visible in moving areas 
depending on the amount of motion. 

Hence, motion compensating interpolation (MCI) techniques have been developed, which are recently 

25 reviewed in [4]. MOtechniques take into account the motion of objects to preserve the natural impression 
of motion. The main difficulty is to provide a sufficiently precise estimation of the motion parameters, in 
order to limit the complexity of the algorithm, most of the MCl-techniques are based on the assumption of 
pure translatorily displaced objects in the image plane [1,5-7], as wen as the presented technique; In [1] and 
[5] the images are subdivided into a fixed number of rectangular blocks. One displacement vector is 

30 determined for each block belonging to a moving area, whereas in [6] only one representative displacement 
vector is determined for each moving area. The algorithm, described in [7], proposes the estimation of one 
displacement vector for each picture element in the moving image parts, to obtain an improved rendition of 
motion in the reconstructed television sequence. 

Summary of the Invention: 

It is an object of this invention to provide a method as described in the preamble of patent claim 1 
which copes with large displacements. 

40 It is another object of this invention to provide uniquely defined displacement vector fields, which are valid 
for the temporal positions of the fields to be interpolated rather than for the transmitted fields. 
A further object of this invention is to present suitable applications of this method. 
The invention is characterised by the features of the claims. 

The invention describes the complete algorithm of a motion compensating field interpolator, that is 

45 applicable for the reconstruction of omitted fields, e.g. in a video conference codec. The hierarchically 
structured displacement estimation technique combined with a change detector copes with large displace- 
ments, which occur even in typical video conference sequences, if several successive fields have been 
dropped at the transmitter. In contrast to the algorithm in [7] this technique provides uniquely defined 
displacement vector fields, which are valid for the temporal positions of the fields to be interpolated rather 

so than for the transmitted fields. This further reduces the jerkiness in the motion compensating interpolated 
sequence. The displacement estimation technique is based on the differential displacement estimation 
algorithm described in [8]. 
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Brief Description of Drawings *i 

The features and advantages of the invention will be more apparent from" the detailed description 
hereunder taken in conjunction with the accompanying drawings, wherein: 

Fig. 1 shows the principle of the motion compensating field interpolator. 

Fig. 2 illustrates the transiatory displacement in television sequences. An object h^s been displaced 
from field k-l to field k by a vector D with the components dx,dy. 

Fig. 3 illustrates several motion compensating iteration techniques for displacement estimation using 
displaced measurement windows. 

(a) measurement window in field k Is displaced. 

(b) measurement window in field k4 is displaced. 

(c) measurement windows are displaced symmetrically against each other. 

Fig. 4 illustrates two distinct measurement windows placed in two successive field k-l (a) and k (b). 
The windows are of size 65 by 65 and 13 by 13 picture elements, respectively. 

Fig. 5 is a 3-D plot of the approximated expectation of the squared displaced frame difference (z- 
axis) versus the displacement vector components dx,dy (x-and y-axis). The results have been obtained by 
displacing the measurement windows placed in the field shown in Fig. 4 (a) to all positions up. to 20 picture 
elements around the shown initial position (x=0,y=0) in the horizontal (y-) and vertical (x-) direction. The 
windows placed in the field shown in Fig. 4 (b) were fixed. 

(a) The windows of 13 by 13 picture elements have been used. 

(b) The windows of 65 by 65 picture elements have been used. The image signal covered by the 
windows has been bandlimited by a FIR filter (RU Table 2) and subsampled by a factor of 4:1 in horizontal 
and vertical direction. 

Rg. 6 illustrates the change detection for the case of a one-dimensional signal. 
Rg. 7 is a block diagram of the change detector. 

Rg. 8 illustrates the change detection mask, obtained by evaluating the frame difference signal 
shown in Rg. 9 (b). Unchanged areas are black displayed. The image contents of the first of two 
consecutive transmitted fields are inserted in the changed areas. 

Rg. 9 illustrates of the motion compensating interpolation filter. The picture element at Xe,y 0 in the 
field to be interpolated is calculated by a weighted sum of the picture elements in field k and k-l, connected 
by the displacement vector D. 

Rg. 10 demonstrates the motion compensating field interpolation for the case of transmitting every 
fourth field. All fields shown, are taken from the temporal centered position between two consecutive 
transmitted fields. The luminance components only are displayed. 

(a) Original field of the sequence Trevor". 

(b) Frame difference signal caused by the luminance changes between the transmitted fields. 
Positive and negative differences are displayed with white and black luminance values, a zero difference is 
gray displayed. 

(c) The field generated by linear interpolation without motion compensation. 

(d) The field generated by motion compensating interpolation. The hierarchically structured 
displacement estimator with the parameters Ksted in Table 1 has been used. 

Rg. 11 demonstrates of the motion compensating field interpolation for the sequence "Split Screen". 
All simulation and display parameters are the same as in Rg. 9 

(a) Original field of the sequence. 

(b) Frame difference signal. 

(c) The field interpolated without motion compensation. 

(d) The field motion compensating interpolated. 

Table 1 shows the parameters for the hierarchically structured displacement estimator as used for the 
simulations. 

Table 2 shows the impulse responses of the digital FIR filters used for the bandlimitation of the image 
signal. 
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Detailed Description of the Invention: 

1. General structure of the interpolator 

5 First the whole structure of the motion compensating field interpolator is outlined, ft consists of three 
blocks: displacement estimator, change detector and motion compensating interpolation filter. These blocks 
are described in detail in Sections 3 t 4 and 5. Experimental results based on the interpolation of more than 
180 fields by means of computer simulations, are discussed in Section 6. 

10 . < .; , . • - 

2. Structure of the motion compensating interpolator 

The motion compensating field interpolator consists of a displacement estimator, a change detector, 
arid a motion compensating interpolation fitter as shown in Fig. 1. The input data is a digital television 
75 sequence containing luminance and chrominance components in fields without Gneinteriace. For the case, 
that fields of a sequence are Bneinteriaced, a vertical filtering of every second field can be appOed to obtain 
a non-interlaced format 

The interpolation algorithm is based on an image model, which is restricted to rigid objects translatorily 
displaced in the image plane. A displacement estimator calculates a displacement vector for each picture 

20 element of a field to be interpolated at a given temporal position between two available fields. A change 
detector distinguishes between changed and unchanged regions of the image contents. This information is 
used to assign zero displacements to the picture elements in unchanged areas. Thus, erroneously non-zero 
estimated vectors in these regions are eliminated. Only the luminance data is used for displacement 
estimation and change detection. 

25 Each displacement vector is determined in the, way, that- it connects two- picture elements of two 
•available fields and crosses the spatial position of the picture element to be interpolated. In the motion 
compensating interpolation filter a weighted sum of the picture elements, connected by the displacement 
vector, is calculated for each picture element of the field to be interpolated. The displacement vectors, 
calculated by means of the luminance data only, are used for both, the interpolation of the luminance 

30 signals and the interpolation of the chrominance signals. 

a Displacement estimator 

35 3.1 The basic estimation algorithm 

The estimation algorithm is based on the assumption of a translatorily displaced rigid object in the 

image plane, that does not change its luminance from field k-l to field k, as shown in Fig. 2 . Then, for a 

moving area the equation 
40 S w (x,y) = Sk(x+dx f y + dy)(1) 

holds, where Sic(x,y) denotes the luminance in field k-l at the spatial position x,y and Sr (x + dx f y + dy) is the 

corresponding luminance, displaced by a vector D with the components dx,dy in field k. Thus, a moving 

object causes a frame difference signal FD, where 

FD(x,y) = Skfry) -S k4 (x,y). (2) 
<5 Compensating the displacement by an estimated displacement vector 6 with the components d x, 3y 

the remaining frame difference, called displaced frame difference DFD, results as 

DFD(x.y,5) = S k (x+ dx,y+ 3 y) -Sk 4 (x,y). (3) . 

Under the above mentioned assumptions, the DFD approaches to zero if the estimate £ is close to the 
true displacement vector D. In [8] an estimation algorithm is derived, that minimizes the local mean squared 
so displaced frame difference. The estimated displacement vector components are determined as 

2x - { E[G x .G y ]*E[FD-G y ] - B[FD *C x ] -e[g~] } / DEN 

h - { e[g x -g ].b[fd-g x ] - e[fd-g ].e[g x ] } / DEN 

55 

with the denominator (4a) 
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DEN [G 2 ] . e [q2 ] - . g*fjr. -G y J, 



5 wfjere the coordinates x,y are omitted for simplicity. The components 
a x(x.y) = {SSfayysx + 5S w (x,y)/5x}A2 
Gy(x,y) = {SSfoyysy + 5S W (x,y)/Sy} / 2 (4b) 

are averages of first order derivatives of the luminance signal of two successive fields with respect to the 
coordinates x and y, respectively. The algorithm given by eq. (4) has been derived using a two-dimensional 

1Q polynomial of second order as an image model for the luminance signal Skfoy) and S^fcy). Due to this 
image model the precision of the estimate is improved compared to other known algorithms [7,9], as shown 
in [8]. Of course, in digital video processing the expectations in eq. (4) have to be approximated by 
.: summing over measurement windows of a certain size. Usually these measurement windows are rectangu- 
lar and an estimation obtained from eq. (4) is then assigned to the center of the window. Also the spatial 

76 gradients have to be approximated by means of the samples of the luminance signal. Adopting a proposal 
-..:■! from [10], the spatial derivatives are approximated as one half of the differences between the two adjacent 
picture elements in the x-and y-direction, respectively. 

It should be noted, that all vector components obtained by eq. (4) are rounded to the nearest integer 
value. Thus, we do not have any non-integral vector component Neither in displacement estimation nor in 

20 motion compensating field interpolation we have to perform any spatial interpolation of picture elements 
between the lattice elements of field M or field k. A signal containing any translatory motion in the image 
plan can be perfectly interpolated using integral displacement vector components, as shown in [11]. 



32. Motion compensating iteration 



A displacement estimate, obtained by evaluating a differential estimation algorithm, is often far away 
from the true displacement even if the present motion is restricted to pure translatory motion. This is due to 
the fact, that the actual image signal differs drastically from the mathematical image model the algorithm is 
30 based on. In this case, the displacement estimate can be improved by a motion compensating iteration of 
the estimation algorithm, as explained in the following. To obtain an estimate for the vector that indicates 
the displacement of an object displaced from an arbitrary position x* y 0 in the field k-l to the unknown 
position XQ+dx f y 0 +dy in field k, the measurement windows are placed in the way, that the centers are given 
by Xo,y e for both fields. The five distinct expectations needed forjsq. (4a) are approximated by summing 
35 over these so placed windows. After calculating the components d x, 5y the estimate can be improved by 
a motion compensating iteration of the algorithm. For that purpose, the frame difference is compensated by 
the vector estimated up to now, i.e. by the vector obtained in the first step^of iteration. This is done by 
displacing the measurement window in field k to the position x,y 0 + ^y and calculating again the 

five expectations. The frame difference FD in eq. (4a) is theji changing into the displaced frame difference 
DFD given by eq. (3) as a function of the components Wx, 3y calculated before. Moreover, all terms 
belonging to field k have to be taken from the displaced position including the spatial derivatives of S k (x,y). 
The displacement vector calculated in this second step Is added to the vector obtained in the first step of 
iteration. This procedure is repeated until a sufficiently precise estimation is achieved. 

This kind of iteration technique, illustrated in Fig. 3 (a), applied to all picture elements of field k-i yields 
4S a displacement vector field that is uniquely defined for field k-l. However, it does not assign a displacement 
vector to all picture elements of field k. To obtain a vector field uniquely defined for field k, the 
measurement windows placed in field k are fixed, those in field k-l are displaced in the motion compensat- 
ing iteration, as shown in Fig. 3 (b). 



3.3 Symmetrized motion compensating iteration 



For motion compensating interpolation of a field at an arbitrary temporal, position between two 
successive fields of a television sequence, a uniquely defined displacement vector field for the field to be 
5g interpolated is needed. Using the iteration technique described in Section 3.2 we obtain a vector field that is 
defined either for field k-l or for field k. Thus we do not have the assignment of displacement vectors to all 
picture elements of the field to be interpolated. In order to overcome this problem a symmetrized iteration 
technique is adopted [12], which is examplarily explained for a field at the centered temporal position 
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between field k-T and k in Rg. 3 (c). In the second Step of the motion compensating iteration both 
measurement windows are displaced against each other. For this example the window in field k is displaced - 
to the position x>+ Jx/2,y 0 + &y/2 and that in field k-i to x,- lx/2,y<r ay/2. Thus, a displace ment vector 
is obtained which is connecting an object displaced from field k*l to field k and which is crossing the spatial 

s position Xo,yo. the vector is to be estimated for. This approach can also be applied to fields at other temporal 
positions between two successive fields. The symmetrized iterafion technique yields a displacement vector 
field, that is uniquely defined for all picture elements of a field to be interpolated at a given temporal 
position.. • • 

In order to avoid any spatial interpolation of picture elements between the lattice elements of the fields, 

70 both measurement windows have to be displaced by an integer number of picture elements in each step of 
iteration. Thus, for the example shown in Rg; 3 (c). the motion compensating iteration by displacing the 
measurement windows symmetrically against each other, always yields displacement vector components, 
that are a multiple of 2 picture elements. At non-centered temporal positions between field k and k-I the set 
of possible values for the displacement vector components is further restricted; For example at the temporal 

is position of one fourth of the field distance to field H, the vector components are. a^ multiple of 4 picture 
elements. To overcome this problem, a special procedure for the motion compensating interpolation of 
three fields between every two given fields is applied, as described in Section 6.2. 

20 3.4 Spatial and temporal recursion 

One can decrease the number of iterations needed to obtain a sufficiently precise estimation by 
recursion techniques. A recursive estimation starts like the second step of iteration as described in Section 
3.2. For a spatial recursion an initial guess obtained by means of displacement vectors previously 

25 calculated for adjacent picture elements serves for the first motion compensation step. Thus, only an update 
term is determined and added to the initial vector. This technique has been investigated by several authors, 
e.g. [13]. A temporal recursion is performed by using tiie displacement vector field determined for the 
previous field of the image sequence. An algorithm described in [14] makes use of this technique. Both, 
spatial and temporal recursion techniques involve the risk of estimation error propagation at object 

30 boundaries and in the case of non-stationary moving objects. 

For the present these kinds of recursive techniques are not considered in the motion compensating 
interpolation algorithm described here. 



35 3.5 Hierarchically structured displacement estimator 

One of the most critical parameters of a differential displacement estimator is the size of the 
measurement window used for the approximation of the expectations. The largest window we can use 
covers the entire image, and we only obtain a reasonable displacement estimation rf the whole picture 

40 contents are displaced as one moving object e.g. in the case of panning. However, for the purpose of 
motion compensating interpolation an estimate locally approximating the present motion at each picture 
element is needed rather than a mean value of displacements caused by several moving objects. On the 
other hand, using very small windows, e.g. of 3 by 3 picture elements, the estimate tends to be unreliable. 
In the case of large displacements there may be no correspondence between the contents of these small 

45 windows, placed in two successive fields of a television sequence. That means, these windows contain two 
absolutely different image parts, and the evaluation of the differential estimation algorithm does not make 
any sense. The differential estimation of the displacement vector for a moving object covered only by one 
of the measurement windows, is impossible. Moreover, using small windows the expectations are approxi- 
mated insufficiently and we are able to cope with displacements of one or two picture elements at the most 

so Thus, large measurement windows are needed to cope with large displacements. On the other hand, small 
windows are required to be sufficiently local adaptive. 

In order to overcome this problem a hierarchically structured displacement estimator has been 
developed. A motion compensating iteration as described in Section 3.2 and 3.3 with large window sizes in 
the first steps of iteration is applied. The image signal is filtered by FIR lowpass filters in these first steps. 

55 From one step to another step of iteration the window size is decreased and the filtering is reduced. The 
first steps of iteration serve to cope with large displacements. The filtering provides an image signal, which 
is matched to the image model the estimation algorithm is based on. Thus the estimate becomes less 
precise in favour of an increased reliability. After motion compensation by means of the vector estimated in 
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the first steps of Iteration, the residual displacement has to be estimated in the next steps -more precisely. 
The smaller the residual displacement the smaller the window sizes can be choosen avoiding the risk of 
estimation .errors. Being dose to the actual displacement to be estimated, the algorithm is evaluated on the 
unfiftered image signal, v 

To Emit the computational complexity in the first steps of iteration the picture signal is spatially 
subsampled according to the bandfimitation achieved by the RR filtering. This can be done in a way, that 
the number of picture elements involved by the measurement windows Is identical in each step, although 
the image part covered by the windows differs from one to another step of iteration. 

In the following" an example Is given to demonstrate the influence of the window size to the reliability of 
the estimate. Fig. 4 shows two successive fields of a television sequence. We consider the estimation of the 
displacement vector for the picture element given by the center of the measurement windows placed at the 
same spatial position in field k-l (Fig, 3.3(a)) and in field k (Fig. 4 (b )). Two distinct window sizes are used, 
one of 13 by 13, the other of 65 by 65 picture elements. As mentioned in Section 3.1 the expectation of the 
squared displaced frame difference is applied as an optimization criterion for the estimate. The algorithm 
uses a spatio-temporal gradient method to estimate the minimum of this expectation. Fig. 5 shows the 3-D 
plot of the optimization criterion versus the displacement vector components dx,dy. The results have been 
obtained by displacing the measurement windows in field k-l horizontally and vertically to all positions up to 
20 picture elements around the initial position. The windows in field k were fixed. The window size of 13 by 
13 picture elements obviously is too small in order to recognize a definite minimum (Fig. 5 (a)). In fact, the 
algorithm results in an estimate J y = 0 in contrast to the actual displacement of about 8 picture elements. 
The result shown in Fig. 5 (b) has been obtained by using the window size of 65 by 65 picture elements. In 
this case, the image signal covered by the windows has been bandlimrted by a FIR filter and subsampled 
by a factor of 4:1 horizontally and vertically. The minimum of the optimization criterion can be identified 
clearly but not precisely. The accuracy of the estimate can be increased in a second step of iteration using 
smaller measurement windows and evaluating the algorithm on the unfiltered image signal. 



4. Change detector 

4.1 Principle of change detection 

The change detector distinguishes between temporally changed and unchanged regions of two 
successive fields. The assignment of picture elements of these fields to the changed area is oftentimes 
wrong, because of scene inherent noise in the unchanged area The aim of the change detector is to decide 
whether the temporal changes are caused by noise or by relevant changes due to motion of objects or 
illumination changes. 

For this purpose known change detection algorithms [2,13] evaluate the frame difference for every 
picture element of the two successive fields. If the frame difference exceeds a given threshold the picture 
element is assigned to be changed, otherwise to be unchanged. Thus, the change detector provides a 
binary mask indicating changed arid unchanged areas of the two successive fields. 

However, this detection is affected by the appearance of noise, i.e. picture elements are assigned to the 
changed region but belong to the unchanged region or vice versa. To overcome this problem, the frame 
differences are summed up over a measurement window and the outcome is compared with a given 
threshold [2,3]. This operation leads to binary masks with changed areas, much larger than effectively 
caused by the moving object or to boundaries, frayed out between unchanged and changed regions. 

Fig. 6 illustrates the change detection using a one-dimensional signal as an example . An object has 
been moved by a displacement dx from field k-l to field k. The absolute frame differences are compared 
with a threshold of zero, assuming a noiseless signal. The resulting change detection mask distinguishes 
between changed and unchanged regions. As shown in Rg. 4.1, this mask borders on the left-hand 
boundary of the object in field k-l and on the right-hand boundary of the displaced object in field k. 



42. The change detection algorithm 

Rg. 7 shows the blockdiagram of the change detector, that determines the frame differences and 
performs a threshold operation, an elimination of singular uncertain elements, a median filtering, and an 
elimination of singular elements. 
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v First the frame difference between field Ic and field k-l Is: calculated, as defined in eq. (2); Evaluating the 
absolute Urame cTrfference frceach picture element independently, one of three states unchanged C,=0, 
changed C,«1, or uncertain C,=X is assigned to each element of an image. Using the uncertain state 
C,=X, misdecisions caused by noise can be avoided in the first processing step. Therefore/ the uncertain 
5 state is trea^ separately in a further operation. 

Picture elements are assigned to the unchanged state, rf the absolute frame difference is below a 
certain threshold T, and to the changed state, if the absolute frame difference is beyond another certain 
threshold T a > T,. The remaining picture elements are assigned to the uncertain state. The selection of the 
thresholds T, and T, has to be adapted to the noise amplitude in the image sequence. /, 
io To obtain a binary mask distinguishing only between changed and unchanged picture elements, an 
t . elimination of singular uncertain elements is performed and non-singular uncertain elements are assigned to 
the changed region. An element is defined to be singular, if this element is of one type, changed or 
unchanged or uncertain, arid at least six direct neighbouring elements are of another type, that means 
unchanged of changed. Thus, each of the singular uncertain elements is assigned either to the unchanged 
75 area, if the neighbouring picture elements are of the changed type or to the changed area, if the- 
^neighbouring picture elements are of the unchanged type. The remaining uncertain elements are assigned 
to the changed region. Now there exists a binary mask distinguishing between changed and unchanged 
elements for each picture element of the two transmitted fields. 

In 1he following step a median filter is applied, using measurement windows of the size of N by N 
20 picture elements. If 1^/2+1 picture elements of the mask belong to the changed area, the picture element 
in the center of the window is assigned to the changed area too or vice versa. This filter can be 
implemented as a simple counting operation. The median filtering smoothes the boundaries between the 
changed and the unchanged area. Further, small regions misdedded in the previous steps are corrected. 
In the last processing step, named elimination of singular elements, still remaining singular elements 
25 are reassigned to the state of the neighbouring elements. 

Rg. 8 shows an example for the change detection mask, obtained by computer simulations. The 
thresholds for the change detector are choosen to T,= 3/256 and T 2 = 6/256, where 256 is due to the 
quantization according to 8 bit per sample. The median filtering is performed using a window size of 5 by 5 
picture elements. The resulting change detection mask is preserving the moving objects in the changed 
30 regions in its entirety and is adapting well to the boundaries of the moving objects. 

5. Motion compensating interpolation filter 

35 The motion compensating interpolation filter calculates each picture element of the fields to be 
interpolated by means of two transmitted fields and by means of the estimated horizontal and vertical 
displacement vector components (Rg. 1 ). Because of the integral displacement vector components (see 
Section 3.1) the filter reduces to a two-coefficient spatio-temporal filter. 

With help of the integral displacement vector components estimated for the position x,y f first the 

40 addresses of the corresponding picture elements in the transmitted fields are determined. They are needed 
to calculate the picture element at the position x,y in the field to be interpolated. The displacement vector 
as well as the two picture elements are multiplied by weighting factors, that correspond to the temporal 
distances between the field to be interpolated and the transmitted fields." Then the picture element to be 
interpolated results from the addition of the two weighted picture elements. 

45 Rg. 9 illustrates the operation of the MCMifter algorithm. The transmitted fields have the associated 
temporal position t=0 for field k-l and t=1 for field k. The intermediate position t=T with 0 £ t £ 1 is 
corresponding to the temporal distance of the field to be interpolated to the field k-l. Each picture element 
of the field to be interpolated is calculated as a function of the displacement vector components and the 
temporal position r as ^ 

so S(x,y.r) = {1-r> S w (x-r* <* x,y-r« 3y) + r»Sk(x-{1-r> 3x,y-{1-r> 3y) 
with (5) 
0£r£1. 

The displacement vectors estimated by means of the luminance data only are also used to interpolate 
the chrominance signals. 
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6. Experimental results 
6.1 Test sequences 

s The presented motion compensating field interpolator has been experimentally investigated by means 
of computer simulations. Two typical video conference scenes with 50 Hz field frequency have been used, 
Trevor" consisting of 150, and "Split Screen" consisting of 100 fields. These sequences have been 
selected by the European COST 21 Ibis simulation subgroup for testing of video conference codecs. The 
original sequences have been sampled at 13.5 MHz for the luminance component (Y) and at 6.75 MHz for 

io each chrominance component (R-Y.B-Y), and uniformly quantized according to 8 bit per sample. For the 
simulations of the interpolator the horizontal resolution of the luminance and chrominance components has 
been reduced to one half of the original sampling rate. Thus, every luminance field; consists of 288 lines and 
312 picture elements per line, the chrominance fields of 288 lines and 156 picture elements per line. Fig. 10 
(a) and 11 (a) show the luminance of one original field for each sequence in the specified format For the 

75 aim of data compression the field frequency is reduced to 12.5 Hz by omitting three fields oufof four fields 
at the transmitter. Effects from line-interleaving have not to be concerned as a result of dropping an odd 
number of successive fields. At the receiver the omitted fields are reconstructed by means of the presented 
motion compensating interpolator. Finally, the reconstructed sequences are converted to the standard line* 
interleaving format by vertical filtering of every second field, in order to display the sequences on a 

20 standard video monitor. 



6.2 Simulation parameters 

25 The problems due to the integer estimation of the displacement- vector components, using the 
symmetrized motion compensating iteration technique at non-centered temporal positions between two 
available fields, are reduced by interpolating the fields in a special order. First, the second of three omitted 
fields is interpolated, by itself at the centered temporal position. Then, the other two fields are interpolated, 
now being at the centered temporal position between one of the originally transmitted fields and that one 

30 interpolated before. The luminance data only is used to calculate the displacement vector fields and the 
change detection masks. The displacement vector fields are applied for both, the interpolation of the 
luminance and of the chrominance fields. 

The parameters for the displacement estimate are listed in Table 1. The impulse responses of the FIR 
filters, used for the bandlimitation of the image signal, are listed in Table 2. The displacement estimation 

35 algorithm is evaluated using the hierarchical structure combined with the symmetrized motion compensating 
iteration technique. Three steps with distinct parameters are used. In each step three iterations are 
performed. The displacement vectors are estimated only for a certain subset of picture elements rather than 
for all spatial positions in each step of the hierarchy. For the other picture elements the vectors are obtained 
by bilinear interpolation, calculating a distance-weighted average of the nearest four vectors belonging to 

40 the rectangular grid, the vector field has been estimated for. The magnitude of the displacement vector 
components is limited to 60 picture elements in the x-and y-direction. 



6.3 Discussion of the results 

45 

The sequences, reconstructed by the presented motion compensating interpolator, have been com- 
pared to results obtained without motion compensation. For this purpose, the sequences have been 
displayed on a video monitor using a real-time image display system. The sequences obtained by linear 
interpolation as well as those, generated by field repetition, show visible degradations. The picture quality is 

so degraded as a result of the relatively large amount of motion. The motion compensating interpolation 
technique yields unblurred moving objects widely preserving the natural impression of motion. In order to 
give an impression of the performance of the proposed approach, several photos taken from the television 
screen are presented as monochrome pictures. 

In Fig. 10 (a) and 11 (a) one original field of the sequence "Trevor" and "Split Screen" is shown. In 

55 order to demonstrate the amount of motion, the frame difference signals between the two transmitted fields, 
used as input data for the interpolator, are shown for both sequences in Fig. 10 (b) and 11 (b). The fields, 
generated for the centered temporal position between the two transmitted fields by means of linear 
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interpolation, are shown In Fig. 10 (c) and 11 (c). Blurring can be recognized in all moving areas. The ] 

results, obtained by using the described interpolator with the hierarchically structured displacement \ 
estimator are shown in Fig. 10 (d) and 11 (d). 



7. Conclusion 

An algorithm for motion compensating field interpolation in digital television sequences is presented. \ 
Linear interpolation and field repetition techniques without motion compensation yield visible degradations 

10 as blurring and jerkiness. In order to avoid these effects, the motion of objects has to "be taken into account 

A hierarchically structured cfisptkjbment ekimatdr is presented, which is able to cope' with large j. 
displacements by means of motion compensating fterafiohs. It is shown, that the evaluation of the \\ 
estimation algorithm on large measurement windows combined with a lowpass filtering of the image signal ! 
in the first steps of the hierajxrfiy increases the reliability of the estimation results. The computational ! 

75 complexity due to the large windows can be decreased by subsampfing the filtered image contents 
according to the bandOmitation. A symmetrized iteration technique enables to provide displacement vector 
fields, which are defined for the fields to be interpolated. Erroneous non-zero vectors in unchanged areas ' 
may cause fitter. They are suppressed by a special change detector. Using these vector fields for the J : 

motion ^compensating interpolation, the rendition of motion is remarkably improved when compared to I 

20 known interpolation techniques. I 
The described interpolation scheme allows the unblurriBd reconstruction of several fields between every j 
two transmitted fields using integer displacement vector components. The evaluation of nearly 200 fields, ! 
interpolated by computer simulations for the case of transmitting only every fourth field of a sequence, 
show that the natural impression of motion is widely preserved. Some remaining visible artefacts are due to 

25 large displacements of more than 60 picture elements or non-translatory motion, which are not sufficiently 

approximated by displacement vectors. ] : 

The remarkably improved picture quality achieved by the presented interpolator has to be payed by a 
relatively complex algorithm, however. 

The method according to the invention can be used for the following purposes: For the reconstruction at the 
30 receiver's end of one or more omitted television images between every two transmitted fields of a digital 

television sequence; for the generation of one or more additional fields between every two successive '{. 
images; for the motion compensating noise reduction of digital television sequences. 
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Table 1 

Parameters for the hierarchically structured displacement estimator as used for the simulations 
WV,WH^ Height and width of the rectangular measurement windows (in picture elements). 
20 F » Name of the filter used for the bandlimitation of the image signal. The impulse responses are listed in 
Table 6.2. 

SF = Factor the image contents of the measurement windows are subsampled by. 
I = Number of iterations performed in each step of the hierarchy. 

GV.GH = Vertical and horizontal distance between the picture elements belonging to the grid the 
25 displacement vector field is estimated for. 



Step of 
hierarchy 




WH 


F 


SF 


I 


GV , 


GH 


1 


65 , 


65 


FIL1 


4 


3 


8 , 


8 


2 


27 , 


27 


FIL1 


4 


3 


4 , 


4 


3 


13 , 


13 


FIL2 


2 


3 


2 , 


2 



Table 2 

Impulse responses of the digital FIR filters used for the bandlimitation of the image signal. 

45 



Filter 


Impulse response 










FIL1 


-13 , -9, 8, 37, 


66, 78, 


66, 


37, 8, 


-9,-13 


FIL2 


13 , -1,-25, -1, 


79,126, 


79, 


-1,-25, 


-1, 13 
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Claims 

1. A method of moticin compensating field interpolation of one or more fields between every two 
transmitted fields of a digital television sequence comprising the steps of. 

5 -gerieraifng a displacement vector by an" iteration process for each picture element of the field to be 
interpolated; v *" 

-assigning zero displacement vectors in unchanged picture areas by means of a change detector, 
characterised in that 

-a hierarchically strobtured dispis^riient estirriation is aippiied to cope with large displacements, wherein 
to after fhotion compensation by means of W vector estimated in the first steps of iteration the residual 
displacement is estimated in the n^xt steps more precisely and wherein the displacement vector is provided 
with integer components, 

-a symmetrized motion compensating iteration is carried out resulting in displacement vectors defined for a 
temporal posifion of the fields to be interpolated in such a way, that an interpolation filter interpolates each 
75 picture element of the fields to be interpolated by means of a displacement vector and a picture element of 
every two transmitted, fields. 

2. A method according to daim 1, wherein in the first steps of the hierarchically structured displacement 
estimation the image signal is low-pass filtered and large measurement windows are applied to estimate 
large displacements, which serve as initial values for the next steps of the iteration process, and wherein in 

20 the last steps of toe hierarchically structured displacement estimation/the unfitered image signal and small 
measurement windows are applied to estimate the displacement vector components locally adaptive. 

3. A method according to claim 1 or 2, in which *the change detector information, processed by a 
threshold operation applied to the absolute frame differences with ah assignment for every picture element 
to belong either to the unchanged area or to. the changed area, is post-processed by means of a median 

25 filter using measurement windows in such a way, that the boundaries between changed and unchanged 
areas of a binary change detection mask is adapted to the boundaries of the moving objects. 

4. Use of the method according to claims 1, 2 or 3 for the reconstruction at the receiver's end of one or 
more omitted television images between every two transmitted fields of a digital television sequence. 

5. Use of the method according to claims 1, 2 or 3 for the generation of one or more additional fields 
30 between every two successive images. 

6. Use of the method according to claims 2 or 3 for the motion compensating noise reduction of digital 
television sequences. 
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